Classification with Costly Features using Deep Reinforcement Learning

We study a classification problem where each feature can be acquired for a cost and the goal is to optimize a trade-off between the expected classification error and the feature cost. We revisit a former approach that has framed the problem as a sequential decision-making problem and solved it by Q-learning with a linear approximation, where individual actions are either requests for feature values or terminate the episode by providing a classification decision. On a set of eight problems, we demonstrate that by replacing the linear approximation with neural networks the approach becomes comparable to the state-of-the-art algorithms developed specifically for this problem. The approach is flexible, as it can be improved with any new reinforcement learning enhancement, it allows inclusion of pre-trained high-performance classifier, and unlike prior art, its performance is robust across all evaluated datasets.Comment: AAAI 201

Family house Rozvadze

Predmetom tejto bakalárskej práce je vypracovanie projektovej dokumentácie k vytvoreniu novostavby rodinného domu v katastrálnom území obce Trenčianske Stankovce, časť Rozvadze. Rodinný dom je samostatne stojaci dvojpodlažný objekt na okraji obce. Objekt je jednogeneračný, navrhnutý pre 4–5 osôb. Súčasťou objektu je garáž s jedným státím. Pôdorysný tvar objektu pripomína písmeno T. Konštrukčný systém je pozdĺžny, stenový. Zvislé a vodorovné konštrukcie sú zo systému HELUZ. Zastrešenie objektu je riešené plochými strechami. Nad časťou objektu je extenzívna zelená strecha. Výkresová časť práce je spracovaná počítačovým programom AutoCAD.The subject of this bachelor thesis is working out the project documentation to execution of a new detached house in cadastral area of Trenčianske Stankovce, part Rozvadze. Family house is detached two-storeyed building, located at the edge of the village. Object is meant for one generation, designed for 4-5 people. The building includes a garage with one car stand. Floor plan is designed in shape of a latter T. Construction system is longitudinal, wall made. Vertical and horizontal supporting structures of the house are made by the system HELUZ. Roofing of the building is designed with flat roofs. On the part of the object is extensive green roof. The drawing part is handled by the AutoCAD software.

Challenges and Open Questions of Machine Learning in Computer Security

This habilitation thesis presents advancements in machine learning for computer security, arising from problems in network intrusion detection and steganography. The thesis put an emphasis on explanation of traits shared by steganalysis, network intrusion detection, and other security domains, which makes these domains different from computer vision, speech recognition, and other fields where machine learning is typically studied. Then, the thesis presents methods developed to at least partially solve the identified problems with an overall goal to make machine learning based intrusion detection system viable. Most of them are general in the sense that they can be used outside intrusion detection and steganalysis on problems with similar constraints. A common feature of all methods is that they are generally simple, yet surprisingly effective. According to large-scale experiments they almost always improve the prior art, which is likely caused by being tailored to security problems and designed for large volumes of data. Specifically, the thesis addresses following problems: anomaly detection with low computational and memory complexity such that efficient processing of large data is possible; multiple-instance anomaly detection improving signal-to-noise ration by classifying larger group of samples; supervised classification of tree-structured data simplifying their encoding in neural networks; clustering of structured data; supervised training with the emphasis on the precision in top p% of returned data; and finally explanation of anomalies to help humans understand the nature of anomaly and speed-up their decision. Many algorithms and method presented in this thesis are deployed in the real intrusion detection system protecting millions of computers around the globe

Symbolic Relational Deep Reinforcement Learning based on Graph Neural Networks

We focus on reinforcement learning (RL) in relational problems that are naturally defined in terms of objects, their relations, and manipulations. These problems are characterized by variable state and action spaces, and finding a fixed-length representation, required by most existing RL methods, is difficult, if not impossible. We present a deep RL framework based on graph neural networks and auto-regressive policy decomposition that naturally works with these problems and is completely domain-independent. We demonstrate the framework in three very distinct domains and we report the method's competitive performance and impressive zero-shot generalization over different problem sizes. In goal-oriented BlockWorld, we demonstrate multi-parameter actions with pre-conditions. In SysAdmin, we show how to select multiple objects simultaneously. In the classical planning domain of Sokoban, the method trained exclusively on 10x10 problems with three boxes solves 89% of 15x15 problems with five boxes.Comment: RL4RealLife @ ICML2021; code available at https://github.com/jaromiru/sr-dr

NASimEmu: Network Attack Simulator & Emulator for Training Agents Generalizing to Novel Scenarios

Current frameworks for training offensive penetration testing agents with deep reinforcement learning struggle to produce agents that perform well in real-world scenarios, due to the reality gap in simulation-based frameworks and the lack of scalability in emulation-based frameworks. Additionally, existing frameworks often use an unrealistic metric that measures the agents' performance on the training data. NASimEmu, a new framework introduced in this paper, addresses these issues by providing both a simulator and an emulator with a shared interface. This approach allows agents to be trained in simulation and deployed in the emulator, thus verifying the realism of the used abstraction. Our framework promotes the development of general agents that can transfer to novel scenarios unseen during their training. For the simulation part, we adopt an existing simulator NASim and enhance its realism. The emulator is implemented with industry-level tools, such as Vagrant, VirtualBox, and Metasploit. Experiments demonstrate that a simulation-trained agent can be deployed in emulation, and we show how to use the framework to train a general agent that transfers into novel, structurally different scenarios. NASimEmu is available as open-source.Comment: NASimEmu is available at https://github.com/jaromiru/NASimEmu and the baseline agents at https://github.com/jaromiru/NASimEmu-agent

Hierarchical Multiple-Instance Data Classification with Costly Features

We extend the framework of Classification with Costly Features (CwCF) that works with samples of fixed dimensions to trees of varying depth and breadth (similar to a JSON/XML file). In this setting, the sample is a tree - sets of sets of features. Individually for each sample, the task is to sequentially select informative features that help the classification. Each feature has a real-valued cost, and the objective is to maximize accuracy while minimizing the total cost. The process is modeled as an MDP where the states represent the acquired features, and the actions select unknown features. We present a specialized neural network architecture trained through deep reinforcement learning that naturally fits the data and directly selects features in the tree. We demonstrate our method in seven datasets and compare it to two baselines.Comment: RL4RealLife @ ICML2021; code available at https://github.com/jaromiru/rcwc